This section is intended to provide a background or context to the invention that is recited in the claims. The description herein may include concepts that could be pursued, but are not necessarily ones that have been previously conceived or pursued. Therefore, unless otherwise indicated herein, what is described in this section is not prior art to the description and claims in this application and is not admitted to be prior art by inclusion in this section.
Scalable Video Coding (SVC) provides scalable video bitstreams. A scalable video bitstream contains a non-scalable base layer and one or more enhancement layers. An enhancement layer may enhance the temporal resolution (i.e. the frame rate), the spatial resolution, or the quality of the video content represented by the lower layer or part thereof. The scalable layers can be aggregated to a single real-time transport protocol (RTP) stream or transported independently.
The concept of a video coding layer (VCL) and network abstraction layer (NAL) is inherited from advanced video coding (AVC). The VCL contains the signal processing functionality of the codec; mechanisms such as transform, quantization, motion-compensated prediction, loop filter, inter-layer prediction. A coded picture of a base or enhancement layer consists of one or more slices. The NAL encapsulates each slice generated by the VCL into one or more NAL units.
Each SVC layer is formed by NAL units, representing the coded video bits of the layer. An RTP stream carrying only one layer would carry NAL units belonging to that layer only. An RTP stream carrying a complete scalable video bit stream would carry NAL units of a base layer and one or more enhancement layers. SVC specifies the decoding order of these NAL units.
The concept of scaling the visual content quality by omitting the transport and decoding of entire enhancement layers is denoted as coarse-grained scalability (CGS).
In some cases, the bit rate of a given enhancement layer can be reduced by truncating bits from individual NAL units. Truncation leads to a graceful degradation of the video quality of the reproduced enhancement layer. This concept is known as fine-grained (granularity) scalability (FGS).
According to the H.264/AVC video coding standard, an access unit comprises one primary coded picture. In some systems, detection of access unit boundaries can be simplified by inserting an access unit delimiter NAL unit into the bitstream. In SVC, an access unit may comprise multiple primary coded pictures, but at most one picture per each unique combination of dependency_id, temporal_level, and quality_level.
Scalable video coding involves the encoding of a “base layer” with some minimal quality, as well as the encoding of enhancement information that increases the quality up to a maximum level. The base layer of SVC streams is typically advanced video coding (AVC)-compliant. In other words, AVC decoders can decode the base layer of an SVC stream and ignore SVC-specific data. This feature has been realized by specifying coded slice NAL unit types that are specific to SVC, were reserved for future use in AVC, and must be skipped according to the AVC specification.
The identification of pictures and their scalability characteristics within an SVC access unit is important at least for two purposes. First, this identification is important for compressed-domain stream thinning in servers or gateways. Due to the requirement to handle large amounts of data, these elements have to identify removable pictures as quickly as possible. Second, this identification is important for the playback of a stream with desired quality and complexity. Receivers and players should be able to identify those pictures in a scalable stream that they are incapable or unwilling to decode.
One function of media-aware gateways or RTP mixers (which may be multipoint conference control units, gateways between circuit-switched and packet-switched video telephony, push-to-talk over cellular (PoC) servers, IP encapsulators in digital video broadcasting-handheld (DVB-H) systems, or set-top boxes that forward broadcast transmissions locally to home wireless networks, for example) is to control the bit rate of the forwarded stream according to prevailing downlink network conditions. It is desirable to control the forwarded data rate without extensive processing of the incoming data, e.g., by simply dropping packets or easily identified parts of packets. For layered coding, gateways should drop entire pictures or picture sequences that do not affect the decoding of the forwarded stream. The interleaved packetization mode of the H.264/AVC RTP payload specification allows for the encapsulation of practically any NAL units of any access units into the same RTP payload (referred to as an aggregation packet). In particular, it is not required to encapsulate entire coded pictures in one RTP payload, but rather the NAL units of a coded picture can be split into multiple RTP packets.
While this liberty of packet aggregation is welcome for many applications, it causes a number of complications in a gateway operation. First, given an aggregation packet, it is not known to which pictures its NAL units belong to before parsing the header of each NAL unit contained in the aggregation packet. Therefore, when the interleaved packetization mode is applied for SVC, the layers in which the contained NAL units belong are not known before parsing the header of each NAL unit in the packet. Consequently, a gateway has to parse each NAL unit header before deciding whether any, all, or some NAL units of the packet are forwarded. Second, for some NAL units, such as Supplemental Enhancement Information (SEI) and parameter-set NAL units, it is not possible to identify the access unit they belong to before video coding layer (VCL) NAL units of the same access unit are received. Therefore, the gateway may need to maintain a buffer and some state information to resolve the mapping of non-VCL NAL units to their associated pictures.
In conventional video coding standards, a picture header is used to separate coded pictures. However, in the H.264/AVC standard and in SVC, no picture header is included in the syntax. Additionally, although parsers may have the ability to parse the scalability information for each NAL unit in a stream, this requires a bit larger amount of processing power, and some parsers may not have this ability.
In addition to the above, an aggregator NAL unit has been previously proposed in the SVC file format verification model 2 (MPEG document M7586). In this system, the aggregator NAL unit is a container that includes the associated NAL units in its payload. The aggregator NAL unit has a type that is unspecified in the H.264/AVC and SVC specifications and must be ignored in H.264/AVC and SVC decoders. However, when a base layer picture according to the H.264/AVC standard is enclosed within an aggregator NAL unit, it no longer is decodable with an H.264/AVC decoder, nor is it parsable with a H.264/AVC RTP depayloadizer or AVC file format parser.